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ABSTRACT 

The Vocational Competency Achievement Tracking System 
(VoCATS) in North Carolina is a competency-based instructional 
management system that allows the collection of data on student 
performance achievement before, during, and after instruction. A 
preliminary pilot study was conducted to evaluate the implementation 
of the VofATS program with secondary business teachers in the western 
part of the state. A survey about VoCATS was then completed by 17 
male and 69 female high school teachers. Approximately 737* were using 
the material? for the first time and nearly 257, were using them for 
the second year. Location of the software varied by school and 
coordinator, but it was generally kept in the high school. 
Keyboarding was the course in which all three VoCATS items were most 
often used. Thirty-nine percent of teachers found VoCATS materials 
very helpful or helpful in planning, and 547. found the curriculum 
guide useful. Forty percent used the VoCATS test-item banks more than 
twice in a year, and 687. used non-VoCATS test materials. 
Seventy- three teachers, if given a choice, would continue to use 
VoCATS materials. Teacher recommendations for improvement centered on 
editing and revising item-bank materials. Six. tables present survey 
findings. (Contains 24 references.) (SLD) 
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INTRODUCTION 

In the 1980's over 30 reports were released concerning education reform 
including The Unfinished Agenda and A Nation at Risk (Lynch, 1991). Weimer (1992) 
and Mahoney (1992) insist the American education sytem is producing functional 
illiterates and reform efforts have failed because of impatience and a search for quick 
solutions. Wirth (1992) in Education and Work for the Year 2000 states that 
vocational education has a vital role in the forming/restructuring of high schools with 
Perkins targeting populations that need assistance. 

Some form of Outcome-Based Education (OBE) is the strategy being used in 
42 states (Varnon and King, 1993) to track student achievement and to be 
accountable at the same time. Outcome-Based Education through redesigning 
student assessment and reporting programs requires students to demonstrate their 
knowledge and skills, thus the assessments used to evaluate student performance 
become critically important (O'Neil, 1993; O'Neil, 1994). Schools should be 
accountable for demonstrating that students have mastered important outcomes. 

Not only is assessment being implemented in vocational education, teachers 
are implementing assessment/accountability/evaluative levels in technology (Madden, 
1993), communication (Goulden, 1992), curriculum (Beale, 1993; Lohman, 1993; 
Sugarman, 1993; Hawkins, et. al., 1993) math (Bulloch and Kent, 1992; Critchley, 
1992), reading and writing (Merina, 1992; Sugarman et. al., 1993), biology (Reid, 
1992), science (Brown, 1992), social studies (Nickell, 1992), geography (Kon and 
Martin-Kniep, 1992), and language arts (Hiebertand Hutchison, 1991). 

The trend toward accountability is probably the most significant development in 
the Human Resource Development (HRD) field in recent years. The hottest issues in 
the HRD field are evaluation and results-oriented training (Phillips, 1991). 
Lee (1990) indicates (HRD) field has grown at a phenomenal rate in recent years in 
formal training-the US organization is spending 45.5 billion dollars on accountability. 
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In addition to Perkins, the impressive work, A Nation Prepared: Teachers for 
the 21st Century (Carnegie Task Force on Teaching as a Profession, 1986) made 
the following statement as one of its 14 recommendations: 

Restructure schools to provide a professional environment 
for teaching, freeing them to decide how best to meet state 
and local goals for children while holding them accountable 
for student progress. 

In the review of literature, no studies directly compare students or schools in 
outcome-based education classes or schools to control groups. Therefore in absence 
of research, communities should decide on what schools and students are 
accountable for. 

Background 

Since the early 1970s vocational education in North Carolina has focused on 
individualized instruction tied to competency-based education. In recent years, 
instructional management techniques and technological developments have evolved 
to a point at which computer-assisted instructional management is now possible. 

Legislative incentives tied to the Basic Education Plan, State Accreditation, and 
Senate Bill 2 have created a demand to structure vocational education locally to 
assess, teach, and track each student's performance (i.e., learning or achievement 
outcomes) based on core competencies. The School Improvement and 
Accountability Act of 1989 created the need to have access to valid and reliable 
information regarding the status of student performance in each vocational course 
and program. 

Obtaining this data in vocational education requires a system which is called 
VoCATS-Vocational Comoetency Achievement Tracking System. VoCATS is a 
competency-based instruc 'onal management system which allows the collection of 
data on st'ident performance achievement prior to, during, and following instruction. 
Data are in the »orm of competency mastery status, pre- and post- test scores and 
gain scores. 

Products included in the VoCATS curriculum package for each vocational education 
course/program are: 

♦ Core competency listing in weighted course/test blueprint format, 

♦ Computer-managed Competency/Test-Item Bank, 
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♦ Curriculum guide with detailed content outline and suggested learning 
activities for students, and 

♦ Competency-based equipment, materials, and supplies list. 



The North Carolina VoCats Survey 

The VoCats Survey of Western North Carolina business teachers was 
developed and completed in response to a SDPI request for proposals in January of 
1993. 

In the past five years, the VoCats components have been implemented in 
several different stages for vocational teachers in North Carolina. Teachers have 
been involved in all of the following tasks: developing and implementing blueprints, 
preparing calendars of instruction, planning and assembling competency-based 
criterion-referenced tests, administering the tests, scoring and generating reports, 
accommodating the at-risk and special needs student, and interpreting and using the 
results of the pre-interim-post tests in teaching and reporting student progress. 

In 1992-93, all Vocational Directors in the state were to have provided their 
vocational teachers with access to copies of the computerized test banks, the 
capacity to tract and record student progress, and available written curriculum guides. 

During the 1992-93 school year, theoretically, all vocational teachers had an 
opportunity to use the blueprints, test item banks and available curriculum guides. 
Feedback from users of the VoCats program is a necessary process before refining 
and revising the existing program. 



Pu rpose and Objectives 

The purpose of this study was to conduct a preliminary pilot study evaluating 
the implementation of the VoCats program with selected secondary business teachers 
in the Western Region. 

Objectives of the study: 

1. To obtain feedback from business teachers concerning the model used 
for implementing the VoCats program in their courses. 

2. To determine the effectiveness of blueprint data on 
class planning and class scheduling. 
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3. To determine the areas of improvement needed in the 
VoCats system. 

4. To develop a survey instrument appropriate for use with all vocational 
teachers within the state. 

5. To provide results of the study to the SDPI, Vocational and Technical 
Education Division. 



METHODS AND PROCEDURES 

A single-page, two-sided survey instrument was developed and reviewed by 
professional educators representing the SDPI, Vocational Directors, High School 
Teachers, and Professional Organizations. Appropriate revisions were made as 
needed. Superintendents, Vocational Directors, and Principals received a copy of the 
questionnaire requesting their permission to survey the teachers in their district. All 
administrators were willing to have their teachers participate in the study. 

Using a list of high school teachers supplied by DPI, a total of 125 
questionnaires were mailed to secondary school business teachers in the Western 
Region of North Carolina. The initial mailing began in late April with a second mailing 
completed in May. 

Returned questionnaires represented a 71 percent response rate. 



FINDINGS 

Respondent Profile 

Respondents were represented by 17 males and 69 females. One respondent 
did not complete the gender identification. Table 1 represents the number of reported 
years of teaching experience for participants. 

High school teachers responding represented a variety of student enrollment 
by school. Table 2 illustrates the size of the high schools from which teachers were 
represented in the study. 
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Table 1 

Number of Years of Respondent Teaching Experience 



Years Number Percent 



Less than One Year 


2 


2.3 


One to Five Years 


23 


26.4 


Six to Ten Years 


7 


8.0 


Eleven to Fifteen Years 


17 


19.5 


Sixteen to Twenty 


12 


13.8 


More than Twenty-one 


26 


29.9 



Teachers reported using the VoCats materials in their business classes as 
follows: 

♦ 63 (72.4 percent) were using the materials for the first time. 

♦ 22 (25.3 percent) were using the materials for the second year. 

No teacher indicated using the materials for more than two years. 



Table 2 

Size of High School by Student Enrollment 



Student Enrollment 


Number 


Percent 


Less than 500 students 


17 


19.5 


500-800 students 


20 


23.0 


800-1200 students 


37 


42.9 


1200-1600 students 


10 


11.5 


More than 1600 students 


1 


1.1 


No response 


2 


2.3 



7 



Person Responsible for Administering VoCats 

Teachers were asked to indicate who was responsible for coordinating VoCats 
within their individual schools. Responses are illustrated in Table 3. 

Table 3 



Person Responsible for Coordinating VoCats 


Title 


Number 


Percent 


Teacher Responding to Survey 


11 


12.6 


Department Chairperson 


3 


3.4 


Vocational Director 


24 


27.6 


IEC 


29 


33.3 


Vocational Coordinator 


6 


6.9 


Other 


14 


16.1 



Titles supplied in the "Other" category included: vocational secretary, 
accounting Teacher, VoCats coordinator (which in one case was another business 
teacher and one indicated the coordinator was available from 11 a.m. -3:00 p.m.), 
and no coordinator. 



Location of VoCats Software 

Respondents were asked "where the copy of the VoCats software was 
located." Two repondents indicated they had their own personal copy, 70 indicated it 
was located within the high school in a variety of locations, 5 indicated the software 
was located at the local administrative unit, and 3 indicated "other." 

The location within the high school included: vocational workroom, computer 
lab, business office, vocational office, business education classroom, career center, 
planning area for voc. building, local director's office, vocational teachers classroom, 
another business teachers classroom, lEC's office, library, guidance office, 
secretary's office, and in a conference room. 



Courses in Which VoCats Materials are Used 

Table 4 illustrates the courses in which respondents are using VoCats 
materials. In the table, BP indicates "blueprint," TIB indicates "test item bank," and 
CG represents "curriculum guide." 
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Keyboarding represents the single curriculum area in which blueprints, test 
item banks, and curriculum guides are most frequently used, After Keyboarding, 
Computer Applications I, Computerized Accounting, Advanced 
Keyboarding/Document Processing, and Office Technology and Procedures I 
represent the four subject areas in which VoCats materials are being used most often 
as indicated by responding teachers. 



How Helpful are VoCats Materials for P l anning Classtime and Lesson Plans? 

Respondents rated the "helpfulness" of blueprints as follows: "very helpful" 
16.3 percent (13); "helpful" 22.5 percent (18); "somewhat helpful" 35 percent (28); 
"not very helpful" 18.8 percent (15); and, "not helpful at all" 7.5 percent (6). 

Sixty-seven of the respondents using curriculum guides rated the "helpfulness" 
of available curriculum guides as follows: "very helpful" 16.3 percent (11); "helpful" 
37.3 percent (25); "somewhat helpful" 32.8 percent (22); "not very helpful" 9 percent 
(6); and, "not helpful at all" 4.5 percent (3). 



Teaching Effectiveness and the Use of Blueprints 

Respondents could check more than one response to the question, " What has 
been the effect of the blueprints on teaching effectiveness?" Results were as follows: 

♦ Mac i lesson planning easier 17.5 percent (22) 

♦ Made objectives for testing 

easier to identify 34.1 percent (43) 

♦ Made it difficult to adhere to 

preset time schedule 28.6 percent (36) 

♦ Increased teaching time for some 

objectives 16.7 percent (21) 

♦ Increased time for teaching 

basic/skills learning tasks 3.2 percent (4) 

What Teachers Did When Topic Was Not Completed During Suggested Time on 
Blueprint 

Teachers were asked to indicate what they did when the blueprint did not allow 

enough time for them to cover a topic. Responses were as follows: 
« 

♦ skip items not covered 17.3 percent (14) 

♦ use time from another unit 60.5 percent (49) 



♦ prepare a handout for items 

missed 11,1 percent (9) 

♦ select items from test bank 

and let students practice 11.1 percent (9) 



Appropriateness of Reading Level for Test Items 

Teachers were asked to indicate how appropriate the reading level of the test 
item banks was for their students. Results for appropriate reading level were as 
follows: "very appropriate lever 3.6 percent (3); "appropriate" 46.4 percent (39); 
"somewhat appropriate" 31 percent (26); "not very appropriate" 16.7 percent (14); 
and, "not appropriate at all" 2.4 percent (2). 



How Often Teachers Used Test Item Banks 

Teachers were asked to indicate how often they used test bank items. The 
responses are illustrated below: 

♦ pre-test and post-test only 27.0 percent (24) 

♦ after each unit 16.9 percent (15) 

♦ every other unit 4.5 percent ( 4) 

♦ more than twice a year 40.4 percent (36) 

♦ other 11.2 percent (10) 



Use of Additional Non-VoCats Test Materials 

Teachers were asked, "How often do you use non-VoCats test materials?" 
Responses were as follows: 

for every unit 33.8 percent (27) 

for most units 33.8 percent (27) 

once or twice a semester 18.8 percent (15) 

never 11.3 percent ( 9) 

other 2.5 percent ( 2) 



Type of Test Questions Used Most Often to Reinforce Class Materials 

Teachers were asked to indicate the type of test question used most often to 
reinforce class material. Responses were as follows: 
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♦ true/false 

♦ multiple choice 

♦ matching 

♦ short answer 

♦ essay 



14 percent (15) 

68.2 percent (68.2) 

12.1 percent (13) 

4.7 percent ( 5) 

.9 percent ( 1) 



Teacher Responses to Writing of Test Items 

Teachers were asked to indicate their opinion of test items. Listed below are 
the responses to the question "generally, test items were:" 

♦ clear and well written 28.8 percent (23) 

♦ testing at the recall 

level only 17.5 percent (14) 

♦ contained errors and 

required editing 53.8 percent (43) 



Assessing Performance Skills 

Responses to the question, "How often did you use test bank items to assess 
performance skills of your students," are as follows: 



after each unit 

every other unit 

occasionally 

more than twice a year 

other 



8.2 percent ( 6) 

2.7 percent ( 2) 

54.8 percent (40) 

15.1 percent (11) 

19.2 percent (14) 



Printing Out VoCats Materials 

More than 58 percent of the teachers indicated they had "very few problems" 
with printing out the test items using the VoCats software. About 30 percent 
indicated they had "a few problems" and 11 percent indicated "quite a few problems" 
or "excessive problems." 



Evaluating Student Progress with VoCats 

Teachers were asked to indicate "how helpful the VoCats program was in 
evaluating student progress." Responses were as follows: 

♦ very helpful 2.8 percent ( 2) 
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♦ 
♦ 
♦ 



helpful 

somewhat helpful 
not helpful at all 



33.3 percent 
40.3 percent 
23.6 percent 



( 9 -4) 
(29) 
(17) 



Teacher Opinion Regarding VoCats 

Teachers were asked to indicate their agreement or disagreement with five 
statements/questions. Table 5 illustrates the opinions of business teachers. 

Overall, more than fifty percent of teacher responses represent positive 
opinions regarding the VoCats program. However, 53 percent of the respondents did 
not feel they had appropriate input into the development of the program. Fifty-one 
percent of the respondents also did not feel that VoCats, as it is now designed, 
measured the learning taking place in the classroom. 



Teacher Comments for Improving VoCats 

Individualized comments were received from 63.9 percent of the responding 
business teachers. A list of original unedited comments is included in the appendix 
of this report. 

An attempt to categorize all the comments provided the following summary of 
results in Table 6. 
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Table 6 



Summary of Teachers Comments to Improve VoCats 
Listed by Frequency of Response 
May 1993 


r,nmmpnt 


Number 


Problems with Test Item Bank-Faulty, irrelevant, outdated, incorrectly 
worded, inability to score short answer cr essay on scantron sheet 


25 


Prlifinrt anri nrnnfroaHinn prrnrQ in V/of^atQ mJltPNJlls 
dulllliy allU pi \J\Jl 1 caUl Iiy ci lUlo in vuvaia uicuci iciio 


9 


Missing or incorrect answer key 


4 


Difficulty with and time required to generate tests without duplication 

of ♦oet ni loctinnc 
UT Icbl LJUcoUUIlo 


4 


Computer malfunctions at individual's school 


3 


Need for more in-service training with test-tracker and test item bank 


3 


Need to coordinate textbooks selected for course and test item banks 


2 


NppH tn consolidate information— too much information to ao through 
in amount of time available. 


2 


NppH fn tpst nerformance skills as well as theorv 


2 


Problems occur with location of software and location of copy 
machine. 


2 


A single coordinator is needed to assist teachers with VoCats 




Teachers need to be allowed to select own pre-test and post-test 
questions. 




Blueprints need to schedule time for testing 




Problems still exist with availability of materials by local 
administration to individual teachers 




Post-testing, using VoCats, was inappropriately timed 




Not sold on benefits of VoCats 




Need equipment to teach course objectives (telecommunications) 
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The following conclusions are based on the findings of this survey. 

1. Respondent profile is representative of the business teacher population 
in Western North Carolina. 

2. VoCats materials had been used by 98.7 percent ofsurvey participants. 

3. No single job title descibes the individual school coordinator responsible 
for administering VoCats Test Item Banks. 

4. Location of the VoCats software varies by school and coordinator and 
may be located within the high school or at a local administrative unit 
office. Eighty-one percent of the respondents indicated that the 
software was located somewhere within the high school. 

5. Keyboarding (which is also most frequently taught) represents the single 
course in which most teachers reported using all three VoCats items, 
blueprints, test item banks, and curriculum guides. With the exception 
of Office Technology and Procedures II, Entrepreneurship and Principles 
of Business, less than 40 percent of the respondents indicated they 
used available curriculum guides (even when they were using test bank 
items and blueprints). 

6. Thirty-nine percent of the respondents found the VoCats materials to be 
very helpful or helpful in planning class time. 

7. Fifty-four percent of the respondents who indicated they were using 
curriculum guides found them to be "Very Helpful" or "Helpful." 

8. Fifty-two percent of the respondents indicated they felt the blueprints 
made lesson planning easier and made testing objectives easier. 
However, 29 percent indicated difficulty in adhering to the time 
schedule. 

9. Sixty percent of the teachers indicated they adjusted the time 
schedule using time from other units when a topic was not completed in 
a unit. 

10. Fifty percent of respondents indicated reading level for test items was 
either "very appropriate" or "appropriate." The other fifty percent still 
have concerns over the appropriateness of reading level for test items. 
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11. Forty percent of the teachers used test item banks more than twice a 
year. 

12. Sixty-eight percent of the teachers indicated they were using non-VoCats 
test materials with their classes. 

13. Multiple choice is the preferred fo/m of test item used by 68 percent of 
the business teachers for reinforcing class material. 

14. Twenty-nine percent of the teachers indicated the test items were clear 
and well-written; however, fifty-four percent reported that test items 
contained errors and required editing before use. 

15. Fifty-five percent of the teachers occasionally used test bank items to 
assess performance skills. 

16. Fifty-eight percent of the business teachers indicated they had "very few 
problems" with printing out test materials. 

17. Thirty-six percent of respondents indicated VoCats program was helpful 
in evaluating student progress. Sixty -four percent indicated it was 
"somewhat helpful" to "not helpful at all." 

18. Fifty-nine percent of the teachers felt critical thinking skills were 
assessed by VoCats test items. 

19. Forty-eight percent of Western North Carolina business teachers felt they 
had appropriate input into the development of the VoCats program. 

20. Seventy-three percent of the teachers responding to the survey indicated 
"if given a choice" they would elect to continue using the VoCats 
materials. 

21 . Forty-nine percent of the teachers indicated they felt competencies in the 
test bank measured the learning taking place in their classrooms. 

22. Seventy-six percent of the teachers feel competency-based programs 
are the most effective way to teach business subjects. 

23. Teacher comments for improving the VoCats program centered chiefly 
on editing and revising the test-item bank materials. 



ERLC 



19 



RECOMMENDATIONS 



16 



1. A Vocational Coordinator needs to be assigned for each individual school 
to oversee the distribution of mat' sis and to assure computer program 
installation and maintenance. 

2. Teachers should be proficient with VoCats software to enable them to 
generate their own tests for each unit within a reasonable time period. 

3. Locating VoCats software adjacent to a copy machine and within easy 
access of all vocational teachers may encourage better use of test bank 
items for classroom practice and additional testing. 

4. Curriculum guides should reflect test bank items and should serve as an 
additional resource fo. the classroom teacher. Where possible, 
curriculum guide material should be written prior to the test-item bank. 
The guides should represent continuit between learning objectives, 
resource materials and test items. Curriculum guides should also be self- 
contained and should require minimal "gathering" or "hunting" on the part 
of the classroom teacher. 

5. Blueprints should include scheduled time for giving unit tests and for 
reviewing test results. 

6. All test-item banks (and answer keys) should be reviewed for correctness, 
reading level, accuracy, relevancy, and format. 

7. More performance skills should be included as part of the pre-test and 
post-test VoCats assessment. Multiple choice questions should not be 
the only test question form used for evaluating student progress by the 
VoCats program. 

8. Continued visibility of the input from NC business teachers must be 
stressed if NC business teachers using the VoCats materials are to feel 
part of the team effort. 



Suggestions for Additionai Research 

1. In the Spring of 1994 the VoCats survey should be administered to all 
business teachers in the state of North Carolina. 
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2. In the Spring of 1994 the VoCats survey, with subject area changes, 
should be administered to all vocational teachers in the state of North 
Carolina. 

3. VoCats classrooms should be pre- and post- tested against non-VoCats 
classrooms to determine if common outcomes for all students represents 
more than minimum competencies for all students. 
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